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(54) Bit rate control for video data 

(57) A bit rate control mechanism for a digital image 
or video compression system estimates a complexity 
parameter for a current picture, or block of samples, of a 
video signal as a function of parameters for a prior pic- 
ture of the video signal, which parameters include a bit 
rate. From the complexity parameter a quality factor for 
the current picture is determined and applied to a quan- 



tizer to compress the current picture. A complexity pre- 
processor may also be used to detect scene changes in 
the video signal prior to estimating the complexity 
parameter. If there is a scene change detected, then the 
rate control mechanism is reset prior to estimating the 
complexity parameter for the first picture in the new 
scene. 
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Description 

Background of the Invention 

5 The present invention relates to data compression, and more particularly to a bit rate control mechanism for digital 
image and video data compression that estimates the number of bits required to represent a digital image or a video at 
a particular quality in compressed form or alternatively estimates the quality achievable for a digital image or a video 
when compressed to a given number of bits, which estimates are used to control the number of bits generated by a 
video compression system. 

10 Visual information may be represented by digital pictures using a finite amount of digital data for still images, and 
by a finite data rate for time-varying images. Such data in its uncompressed form contains a considerable amount of 
superfluous information. Image compression techniques attempt to reduce the superfluous information by minimizing 
the statistical and subjective redundancies present in digital pictures. Pulse code modulation, predictive coding, trans- 
form coding, interpolative/extrapolative coding and motion compensation are some of the tools used in image compres- 

15 sion techniques. 

A digital video/image compression technique may be either lossy or lossless. The lossy compression techniques 
introduce an irreversible amount of distortion into the picture data. In these techniques a trade-off is made between the 
amount of distortion added to the original picture input and the number of bits the compressed picture occupies. A rate 
controller in a video/image compression system controls the number of bits generated by altering the amount of distor- 
20 tion added to the original input by the compression system. In other words a rate controller in a video/image encoder 
controls the number of bits needed to represent the compressed image by changing the quality of the decompressed 
image. 

Transform coding techniques take a block of samples as the input, transform this block into a number of transform 
coefficients, quantize the transform coefficients, and variable or fixed length encode the quantized transform coeffi- 

25 cients. The input to the transform coding system may be either the original picture elements (pixels), such as in JPEG 
and intra-MPEG, or the temporal differential pixels, such as in inter-MPEG. An adaptive still image coding technique 
using a transform coder with a rate controller is shown in Fig. 1. An input image block is transformed by a discrete 
cosine transform (DCT) function, quantized and variable length coded (VLC). The rate controller observes R(n-1), the 
number of bits generated by the previous block, and selects a quantizer scale factor Q(n) for the current block. A still 

30 image coding scheme, such as JPEG, may be used on a motion picture, as shown in the simplified block diagram of 
Fig. 2. In these schemes the rate controller observes R(n-1), the number of bits generated by the previous frame (field), 
and selects a quantizer scale factor Q(n) for the current frame (field). A simplified block diagram of an MPEG encoder 
is shown in Fig. 3, where R(n-1) is the typical number of bits generated in the previous macroblock. For JPEG Q(n) is 
referred to a qfactor or quality factor, and for MPEG it is referred to as mquant. 

35 In all of the schemes shown in Figs. 1-3 Q(n) is used to scale the step sizes of the quantizers of transform coeffi- 
cients (quantizer matrices). Increasing Q(n) reduces R(n) and vice versa. Q(n) is selected so that R(n), the number of 
bits generated with this quantizer scale factor Q(n), is close to the targeted rate for the block, frame or field. Q(n) also 
is an indication of the quality of the decoded block, frame or field. To perform efficiently, a rate control algorithm requires 
a good estimate of the rate-quality relationships for the input data, i.e., R(n) vs. Q(n). A good rate controller would come 

40 up with a Q(n) that results in a targeted R(n). The targeted R(n) for a block, frame or field could vary with n. For example 
it might take into account the visual characteristic of the block in question, whether the coding is variable bit rate (VBR) 
or constant bit rate (CBR). A good rate controller tries to keep the Q(n) smooth over n so that the resulting quality of the 
decoded picture is smooth as well. 

Given actual R(n-1), the actual bits generated for the preceding block number n-7, Chen et al, as described in 

45 "Scene Adaptive Coder" from IEEE Trans. Communications Mar. 1984, compute Q(n) in the following manner. A buffer 
status B(n-1) after coding block n-1 is recursively computed using 

B(n-1) = B(n-2) + R(n-1)-R 

so where R is the average coding rate in bits per block. From the buffer status B(n-1) the quality factor Q(n) is computed 
through 

Q(n) = (1-y)M>(B(n-1)/B) + Y *Q(n-1) 

55 where <H } is an empirically determined normalization factor versus buffer status curve and B is the rate buffer size in 
bits. This produces a smoothly varying Q(n) depending on y. y is taken to be less than unity. 

Alternatively the Test Model Editing Committee, International Organisation for Standardisation, Test Model 3 
(Draft), Dec. 1992 computes Q(n) in a similar way as follows. First the virtual buffer status B(n-1) is computed as above. 
Then Q(n) is computed through the linear relation 



2 



EP0 778 708 A2 



Q(n) = K R *B(n-1) 

where K R is a constant that depends on the targeted average bit rate. This Q(n) may be further scaled based on the 
visual complexity of the block being coded. 
5 Using these techniques Q(n) could change rapidly, and there is no estimate of the quality achievable for a particular 
block, frame or field with a given number of bits. What is desired is a rate control mechanism that estimates the quality 
achievable for a digital image or video when compressed to a given number of bits or alternatively estimates the number 
of bits required to represent a digital image or video at a particular quality in a compressed form. 

10 Summary of the Invention 

Accordingly the present invention provides a bit rate control mechanism for video data compression that either esti- 
mates the number of bits required to represent a digital image or video at a particular quality in a compressed form or 
estimates the quality achievable for a digital image or video when compressed to a given number of bits. A quantizer for 

15 compressing the transform coefficients for a current block of samples of a video signal is controlled by a quality factor 
that is a function of a bit rate for a prior block of samples of the video signal as determined by a rate controller. In the 
rate controller a complexity parameter is determined as a function of the prior block of samples including the bit rate. 
The complexity parameter is then used together with the bit rate to generate the quality factor. The rate controller may 
also include a scene detector for resetting the rate controller at the beginning of each scene. 

20 The objects, advantages and novel features of the present invention are apparent from the following detailed 
description when read in light of the appended claims and attached drawing. 

Brief Des cri p tion pf the Drawing 

25 Fig. 1 is a block diagrammatic view of an adaptive still image coding technique with a bit rate controller according 
to the prior art. 

Fig. 2 is a block diagrammatic view of a motion JPEG scheme with bit rate control according to the prior art. 
Fig. 3 is a block diagrammatic view of an MPEG encoder with bit rate control according to the prior art. 
Fig. 4 is a block diagram view of a bit rate controller according to the present invention. 
30 Fig. 5 is an illustrative view of picture grouping for an overlapping window method of determining quality and tar- 
geted number of bits according to the present invention. 

Fig. 6 is an illustrative view of picture grouping for a non-overlapping window method of determining quality and tar- 
geted number of bits according to the present invention. 

Fig. 7 is a graphic diagram view of buffer occupancy projection for constant bit-rate operation according to the 
35 present invention. 

Fig. 8 is a block diagram view of a complexity pre-processor for determining scene cuts according to the present 
invention. 

Description of the Preferred Embodiment 

40 

The relationship between the quality factor Q of a compressed video and the average bits R generated by a block, 
frame or field of samples is modeled through 

R = a*Q" P t Q>0, a£0, p>0 

45 

where a gives an indication of the complexity of the block being compressed, which may vary from block to block 
(frame/field to frame/field), and p, which empirically has significantly less variations, may be treated as a constant. This 
model is applicable to a number of image and video compression techniques, including JPEG, MPEG and MPEG-2. 
The quality factor Q may be used to generate the qfactor in JPEG or mquant in MPEG through simple scale and satu- 
50 ration operations. 

If a and p for block n equal a(n) and p(n) respectively, the targeted bits R(n) for block n may be achieved by using 
a quality factor Q(n) given by 

Q(n) = (R(n)/a(n))' 1/p(n) . 

55 

In general a(n) and p(n) are not known in advance, but p(n) may be assumed to be a constant p. Then the quality factor 
is given by 

Q(n) = (R(n)/a(n))- 1/p . 
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In motion JPEG and all MPEG coding schemes all the pictures in the video are compressed the same way, and 
only one complexity metric needs to be maintained. An input video signal is input to an MPEG or motion JPEG encoder 
12 as shown in Fig. 4 to obtain an actual R(n-1) for the prior frame. The actual R(n-1) is input to a processor 14. The 
processor 14 has a complexity processor 16 which uses the previous history to estimate a(n): 

5 

a(n) = (1 - Y)*R(n-1)*Q(n-1) p + Y *a(n-1) 

where y is the smoothing factor in the estimation of a, R(n-1) is the actual number of bits used for picture n-1. Depend- 
ing upon the application, a value for y is selected from the range 0^1 . If y = 1 , a(n) is a constant with respect to n, and 

10 if y = o, a(n) depends only on the preceding block coding results. Once a(n) is estimated, then it is input to a quality 
processor 18 where the quality factor Q(n) may be computed as above. In MPEG the average rate R is used to obtain 
Q(n), as well as the targeted R(n) for a particular picture. Q(n) is used to obtain results in actual R(n), which is used for 
updating cc(n). In motion JPEG, as well as in all-l MPEG, targeted R(n) is the same for every picture, i.e., equal to the 
average required rate R. This R is used to obtain Q(n), which is used to obtain actual R(n) for updating o(n). 

75 In a more general compression of video using MPEG the coded pictures may be categorized into three types: I, B 
and P. An Intra-coded (I) picture is coded using information only from itself. A Predictive-coded (P) picture is coded 
using motion compensated prediction from a past" reference frame or past reference field. A Bidirectionally-coded (B) 
picture is coded using motion compensated prediction from a past and/or future reference frame(s). A given picture 
(field/frame) of video has a different coding complexity depending upon whether it is coded as an I, B or P picture. 

20 Therefore three picture complexity measures are used for the video, a,, oq and a P for I, B and P pictures respectively. 
Upon compressing the picture n-1 with a quality factor Q(n-1), the actural output bits R(n-1) are measured. Then 
depending upon the coded picture type t(n-1) the corresponding picture complexity is updated: 

«t(n-i)(n) = (1-YrR(n-1)*Q Vl)+Y*a t(n . l} (n-1) 

25 

The other two picture complexities remain unchanged: 

a s (n) = a s (n-1),S£ (l,B,P}\t(n-1) 

30 Then the target number of bits R(n) and the quality factor Q(n) for the current picture n may be computed through one 
of two methods: overlapping window method and non-overlapping window method. In both methods, as usually done 
in the MPEG world, the assumptions are: 

Q B = K B *Q I 

35 

Qp = Kp*Q, 

where K B and K P are known constants, and Q,, Q B and Q P are the quality factors used for I, B and P pictures respec- 
tively. 

40 In the overlapping window method, also known as the sliding window method, the stream of pictures (fields/frames) 
to be compressed, in coding order as opposed to the display order, are blocked into overlapping windows of size N as 
shown in Fig. 5. In this method pictures 0 through N-1 form the first window (WINDOW 0), pictures 1 through N form the 
second window (WINDOW 1), etc. After compressing each picture, the window is moved to the right by one picture. If 
N,, N B and N P represent the number of I, B and P pictures remaining in the cunent window, then for the overlapping 

45 window method 

N, + N B + N P 3N 
so E(n) & TargetedR(n) - ActualR(n) 

E(-1) = 0 

55 Q,(n) =((0^, + a B N JC* + QpNpKp* J/tfNj + N B + N P )*R +E(n-1))) 1 * 

where R is the average coding rate in bits per picture. From Q, values of Q B and Q P may be computed. 
Finally the target rate R(n) for the picture n is computed through 
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TargetedR(n) = a t(n) *Q l(n) ' p (n) 

where t(n) is the coding type of picture n. 

In summary the overlapping window method has the following steps: 

5 

1 . Initialize: E(-1) <- 0; select values for a's, p, y and N; n <- 0 

2. Before coding picture n 

10 (a) update N,, N B and N P 

(b) compute Q, 

(c) compute Q B or Q P if needed 

15 

(d) compute the target rate R(n) 

3. After coding picture n with a quality factor Q x ^, measure the actual bits generated by picture n 
20 4. Compute E(n) <-TargetedR(n) - Actual R(n) 

5. Update a's 

6. Move the window by one picture, increment n, and go to step 2 

25 

In the non-overlapping window method the stream of pictures to be compressed, in coding order rather than display 
order, is blocked into non-overlapping segments or windows of a preselected size N, as shown in Fig. 6. Each picture 
belongs to one and only one window. Then pictures 0 through N-1 form the first window, pictures N through 2N-1 form 
the second window, etc. If WinBits represents the bits available to the remaining pictures in the window and N,, N B and 
30 N P represent the number of I, B and P pictures remaining in the current window, then for the non-overlapping windows 
method 

N, + N B + N p £ N 

35 and Q(n) and R(n) are computed as follows: 

1 . Initialize: WinBits - 0; select values for a's, p, y and N; n <- 0 

2. Beginning of window: WinBits <- WinBits + N*R 

40 

3. Before coding picture n 

(a) update N,, N B and N P 
45 (b) compute Q,(n) =((a,N, + a B N B K B ~ p + a p N p K p * p )/WinBits) 1/p 

(c) compute Q B or Q P if needed 

(d) compute the target rate R(n) 

50 

4. After coding picture n with a quality factor Q t(n) , measure the actual bits generated by picture n 

5. Update WinBits - WinBits - Actual R(n) 
55 6. Update a's 

In MPEG one of the requirements for generating a correctly coded bitstream is that the Video Buffer Verifier (VBV) 
is not violated. The VBV is a hypothetical decoder, described in ISO/IEC 13818-2 Annex C, which is conceptually con- 
nected to the output of an MPEG encoder. The VBV has an input buffer known as the VBV buffer of size B max bits. The 
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target rate R(n) computed in step 2(d) above in the overlapping window method, or in step 3(d) in the non-overlapping 
window method, may have to be adjusted so as not to overflow or underflow the VBV buffer The occupancy of the VBV 
buffer for a constant bit-rate operation of MPEG is shown in Fig. 7 in idealized form. The VBV buffer occupancy B is 
updated recursively as follows: 

5 If Ba(n-1) is the buffer occupancy right after decoding picture (n-1), the buffer occupancy just before decoding pic- 

ture n, Bb(n). is given by 

Bb(n) = Ba(n-1) + R 

10 where R is the average bits per picture. 

The occupancy Ba(n) just after decoding picture n is given by 

Ba(n) = Bb(n) - R(n) 

75 where R(n) is the number of bits used for picture n. 

The relationship between the number of bits per picture, R(n), and the quality factor Q(n), described above may be used 
by an MPEG encoder to 

1 . maintain the constraints imposed by the VBV, 

20 

2. keep the VBV buffer occupancy operating point center, i.e., away from being nearly full or empty, 

3. enable VBV buffer occupancy terminal conditions to be achieved, and 
25 4. predict and avoid any potential VBV overflow and underflow condition. 

To use this VBV based rate control strategy, the encoder keeps track of the following: x 

1 . the current VBV buffer occupancy at picture n in coding order just before it is removed from the VBV buffer, i.e., 
30 Bb(n), 

2. the number of pictures of each picture type (I, P and B) remaining in the current window, 

3. the target VBV buffer occupancy at the end of a window, this occupancy being the VBV buffer occupancy just 
35 before the last picture within the window is removed from the VBV buffer, i.e., Bb(n + N, + N p + N B ),and 

4. the average number of bits per picture, R, assuming a constant bit rate coding. 

The number of available bits to code all pictures remaining in the window, either overlapping or non-overlapping meth- 
40 ods, is given by 

WinBits = Bb(n) + (N, + N p + N B )*R - Bb(n + N , + N p + N B ). 
Using the model described above the quality factor Q(n) for the remainder of the window is estimated by 

45 

Q ,(n) = ((a ,N , + a P N P K p " p + a B N B K B ~ p )/WinBits) 1/p . 
Then the target bits for each picture type within the window are given by: 
so TargetR, = a,Q," p , 

TargetR P = a P K P * P Q P " p , and 
TargetR B = a B K B P Q B p . 

55 

Using these target sizes for each picture type, simulated VBV buffer occupancy trajectory over the window may be com- 
puted, i.e., Bb(n) and Ba(n)'s for all remaining pictures of the window are projected. If the trajectory indicates a VBV 
buffer overt tow or underflow or comes close to causing the overflow or underflow, then the window is shortened such 
that it ends at the point where the overflow or underflow was indicated. A target VBV buffer occupancy is chosen such 
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that no overflow or underflow occurs. With the shortened window Q,(n) and TargetR's are recomputed. This is shown in 
Fig. 4 where the quality factor Q(n) is input to a buffer occupancy predictor 32 to project the Bb's and Ba's, which are 
then input to a VBV comparator 34. 

When a satisfactory VBV buffer occupancy trajectory is obtained, then the current picture is coded. When the cod- 
5 ing is completed, the acutal size of the picture is then used to update the complexity estimates for the current picture 
type: 

«t ( n-i)(n) ■ (1"Y)*R(n-1)*Q p (n-1) + Y*« t(n . 1} (n-1). 

w For cases where there is no a priori target VBV buffer occupancy to terminate the window, the size of the window is 
chosen such that it ends on a "Group of Pictures" boundary. In this case Bb(n + N, + N P + N B ) is chosen to be: 

Bb(n + N, + N P + N B ) = 0.5*(B ^ + TargetR ,). 

To compensate for the fact that the a parameter does not adapt in a relatively fast manner at scene changes in the 
input video, the input video as shown in Fig. 4 also is input to an activity estimator 20. The detected activity is input to 
comparator22~to determine whether there has been an abrupt change corresponding to a scene change. The activity 
estimator 20, as shown in more detail in Fig. 8, measures the activity £ of the picture to be coded in determining the 
complexity of the picture being compressed. A picture (field/frame) to be coded is broken into four bands by a subband 
analyzer 24. The variance of the energy in the low-high (LH) and high-low (HL) bands is determined by appropriate var- 
iance computational circuits 26, 28, and the two variances are input to a multiplier 30. The measure of activity £ is cal- 
culated as the energy product in low-high and high-low bands: 

v 2*2 

C = a LH a HL 

Any abrupt changes in C from picture to picture indicate a scene change in the video signal. When a scene cut or 
change is detected, the comparator 22 provides a signal to the processor 14 to flush the old value(s) of a(n) and y is 
temporarily made equal to 0, i.e., the system is reset. Other forms of scene cut detections are possible and may be used 
with the rate control mechanism of the present invention. 

Thus the present invention provides a rate control mechanism for video compression that uses a special relation- 
ship model between the quality factor and the average bits generated using an indication of complexity of the block 
being processed. 

Claims 

35 

1 . A bit rate controller for a moving image data compression encoder comprising: 

means in response to an actual bit rate for a prior coded block of the moving image data for generating a com- 
plexity factor as a function of a prior complexity factor and a prior quality factor for the prior coded block; and 

40 

means for computing a quality factor for a present block of the moving image data compression decoder as a 
function of the complexity factor and a target bit rate for the present block. 

2. The bit rate controller as recited in claim 1 further comprising: 

45 

means for projecting a virtual buffer occupancy as a function of the quality factor; and 

means for controlling a width of a processing window for the generating means so that the virtual buffer occu- 
pancy does not overflow or underflow. 

so 

3. The bit rate controller as recited in claims 1 or 2 further comprising: 

means for estimating an activity factor between images of the moving image data; and 

55 means for indicating a scene change to initialize the generating means when the activity factor indicates an 

abrupt change between images. 

4. A method of bit rate control for digital image and video compression of the type wherein a quality factor for control- 
ling a quantizer for a current block of samples of a video signal is generated by a rate controller as a function of a 
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bit rate for a prior block of samples of the video signal comprising the steps of: 

estimating a complexity parameter for the current block of samples as a function of a prior block of samples 
including the bit rate; and 

determining from the complexity parameter the quality factor for output from the rate controller. 

5. The method as recited in claim 4 further comprising the steps of: 
detecting a scene change in the video signal; and 

resetting the rate controller prior to the estimating step when the scene change is detected by the detecting 
step. 

6. The method as recited in claims 4 or 5 further comprising the steps of: 

projecting a virtual buffer, occupancy as a function of a prior occupancy and an average bit rate; and 

controlling a processing window for the estimating step so that the virtual buffer occupancy does not overflow 
or underflow. 

7. A method of compressing moving image data comprising the steps of: 

initializing a bit rate controller by selecting a complexity value and a size for a processing window in terms of a 
25 given number of blocks of the moving image data; 

determining by the bit rate controller a quality factor for a block of the moving image data as a function of the 
complexity value, the size of the processing window, an average bit rate and a bit error rate; 

30 computing a target bit rate for the block of the moving image data as a function of the quality factor and the 

complexity value; 

coding the block of the moving image data in an encoder using the quality factor; 

35 updating the bit error rate as a function of an actual bit rate from the encoder for the block of the moving image 

data and the target bit rate, and updating the complexity value for a next block of the moving image data as a 
function of the quality factor and complexity value for the block of the moving image data; 

moving the processing window by one block of the moving image data; and 

40 

repeating the determining, computing, coding, updating and moving steps for the next block of the moving 
image data. 

8. A method of compressing moving image data comprising the steps of: 

initializing a bit rate controller by selecting a complexity value and a size for a processing window in terms of a 
given number of blocks of the moving image data; 

determining a number of bits for the window as a function of the size of the processing window and an average 
bit rate; 

computing by the bit rate controller a quality factor for a block of the moving image data as a function of the 
complexity value, a remaining size of the processing window and the number of bits for the processing window, 
and computing a target bit rate for the block of the moving image data as a function of the quality factor and the 
complexity value; 

coding the block of the moving image data in an encoder using the quality factor; 

updating the number of bits in the processing window as a function of an actual bit rate from the encoder for 
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the block of th e moving image data, the size of the processing window to determine the remaining size, and 
the complexity value for a next block of the moving image data within the processing window as a function of 
the quality factor and complexity value for the block of the moving image data; 

repeating the determining, computing, coding and updating steps for each block within the processing window; 
and 

moving the processing window by the size of the processing window and repeating the above steps for a next 
group of blocks of the moving image data. 
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